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Modeling  Human  Syllogistic 
Reasoning  in  Soar 

Thad  A.  Polk  and  Allen  Newell 

Department  of  Computer  Science, 

Carnegie  Mellon  University 

Soar  is  an  architecture  for  general  intelligence,  which  has  been  shown  to  be  capable  of  supporting  a 
wide  variety  of  intelligent  behavior  involving  problem-solving,  learning,  designing,  planning,  etc.  (Laird. 
Newell  Sl  Rosenbloom,  1987,  Steier,  eL  aL.  1987).  Soar  has  also  been  put  forth  as  a  unified  theory  of 
human  cognition  (Newell,  1987).  We  provide  support  for  this  by  presenting  a  theory  of  syllogistic 
reasoning  based  on  Soar  and  some  assumptions  about  subjects’  knowledge  and  representation.  The 
resulting  theory  (and  system,  Syl-Soar/S88)  is  plausible  in  its  details  and  accounts  for  existing  data  quite 
well. 


The  Task 


Syllogisms  are  reasoning  tasks  consisting  of  two  premises  and  a  conclusion  (Figure  1,  left).  Each 
premise  relates  two  sets  of  objects  (x  and  y)  in  one  of  four  ways  (Figure  1,  middle),  and  they  refer  to  a 
common  set  {bowlers).  A  conclusion  states  a  relation  between  the  two  sets  of  objects  that  are  not 
common  ( archers  and  canoeists)  or  that  no  valid  conclusion  exists.  The  three  terms  x.yx  can  occur  in 
four  different  arrangements,  called  figures  (Figure  1,  right),  producing  64  distinct  syllogisms. 


Premise  1:  No  archers  are  bowlers  A:  All  x  are  y 

Premise  2:  Some  bowlers  are  canoeists  I:  Some  x  are  y 

Conclusion:  Some  canoeists  are  not  archers  E:  No  x  are  y 

O:  Some  x  are  not  y 

Figure  1:  Syllogism  task. 


I  PI  xy  I  PI  yx  I 
1  P2  vz  I  P2  vz  I 
I  PI  xy  I  PI  yx  I 
I  P2  zv  I  P2  zv  I 


Syllogisms  have  been  much  studied  (see  Joimson-Laird  1983  for  review).  The  essential  problem  has 
been  to  understand  why  some  syllogisms  are  so  hard  while  others  are  so  easy.  However,  the  area  is  also 
useful  as  a  testbed  for  cognitive  theories. 


The  Soar  Theory  of  Syllogisms 

The  Soar  architecture  has  the  following  features: 

1.  Problem  spaces.  All  tasks,  routine  or  difficult,  are  formulated  as  search  in  problem  spaces. 
Behavior  is  always  occurring  in  some  problem  space. 

2.  Recognition  memory.  All  long-term  knowledge  is  held  in  an  associative  recognition 
memory,  realized  as  a  production  system. 

3.  Decision  cycle.  All  available  knowledge  is  accumulated  about  the  acceptability  and 
desirability  of  problem  spaces,  stales  and  operators  for  the  current  total  context,  and  the  best 
alternative  is  chosen  among  those  that  are  acceptable. 

4.  Impasse  driven  subgoais.  Incomplete  or  conflicting  knowledge  at  a  decision  cycle 


POLK,  NEWELL 


produces  an  impasse.  The  architecture  creates  a  subgoai  to  resolve  the  impasse.  Cascaded 
impasses  create  a  subgoai  hierarchy. 

5.  Chunking.  The  experience  in  resolving  impasses  continually  becomes  new  knowledge  in 
recognition  memory,  by  means  of  constructed  productions  (chunks). 

6.  Annotated  models.  States  are  represented  as  annotated  models  (to  model  human 
cognition). 

Figure  2  indicates  the  structure  of  the  system:  the  collection  of  problem  spaces  (triangles)  with 
operators  and  states.  Subspaces  arise  from  impasses,  usually  reflecting  the  need  to  implement  operators 
or  satisfy  operator  preconditions.  The  task  data  structures  occur  in  working  memory  and  are  continually 
viewed  by  the  recognition  memory,  which  contains  all  task- implementation  and  search-control 
knowledge.  Relevant  knowledge  accumulates  from  this  memory,  permitting  steps  to  be  taken  in  the 
current  space  or,  upon  impasses,  creating  subgoals  to  be  solved  in  subspaces,  etc.  The  micromechanics 
are  beneath  the  level  of  detail  of  this  paper,  but  drive  the  entire  system,  including  learning. 


Figure  2:  The  structure  of  Soar. 


A  key  assumption,  developed  strongly  by  Johnson-Laird  (1983),  is  that  humans  represent  the  situations 
presented  in  syllogisms  as  models.  A  pure  model  is  a  representation  that  satisfies  the  structure 
correspondence  condition:  specified  parts  and  relations  of  the  representation  data  structure  correspond  to 
parts  and  relations  of  the  situation,  without  completeness  (see  also  Levesque,  1986).  A  pure  model 
admits  highly  efficient  match-like  processing,  but  is  limited  in  its  representational  power.  An  annotated 
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model  is  a  representation  that  makes  principled  exceptions  to  a  pure  model,  which  increase  its 
representational  power,  while  preserving  essential  match-like  processing.  An  annotation  attaches  to  a 
data-stmcturc  part,  asserting  a  variant  interpretation  for  the  part  (e.g„  not  asserts  that  the  part  is  not  to  be 
found  in  the  situation  where  the  correspondence  mapping  would  otherwise  locate  it).  The  annotations 
used  for  syllogisms  are  not,  optional,  many,  target  and  source.  Annotations  can  quantify,  but  are  local 
and  do  not  admit  unbounded  processing.  Figure  3  (left)  indicates  the  models  that  might  be  built  from  two 
premises.  The  line  through  the  bowling  pin  indicates  a  not  annotation. 


Input  premises: 

PI  Some  archers  are  not  bowlen 
P2  All  canoeists  are  bowlen 


Representation  of  premises 


Syllogism  Space 
Comprehend  Space 


Build-coaclusion 


Prop-to-prop 


Modei-to-prop 


Prop-to-mode! 


read-premise,  build-conclusion 

ail,  some,  no,  are,  not,  and  one  for 
each  generic  noun 

generate -quantifier, 
generate -predicate, 
generate  subject 

auend-to-prop,  copy-subject, 
copy-object. 

copy-sign,  copy-quantifier, 
create-auxiliary 


attend- to -object, 
augment-proposition 

auend-to-prop,  augment-model 


Figure  3:  Annotated  models,  problem  spaces  and  operators  for  syllogisms 

Reasoning  occurs  by  generating  models  to  correspond  to  situations,  inspecting  the  models  for  the 
properties  of  the  situation,  and  forming  new  propositions  to  assert  the  result  Inspection  is  a  power  of  the 
recognition  memory  (production  match).  Since  models  are  limited,  some  situations  can  be  represented 
only  by  a  disjunctive  set  of  models;  reasoning  then  includes  generating  sets  of  models  to  test  conjunctive 
properties.  Reasoning  with  multiple  models  occurs  in  humans  and  has  been  central  to  model-based 
theories  of  syllogistic  reasoning  (Johnson-Laird,  1983,  Inder,  1986),  but  the  present  theory  includes  only 
reasoning  with  a  single  model. 

Six  problem  spaces  are  used  in  syllogistic  reasoning  (Figure  3  lists  them,  with  operators.  Figure  2 
shows  how  they  link  together).  Comprehend,  Syllogism  and  Build-conclusion  form  the  top-level  path 
between  the  presented  premises  and  the  response,  the  knowledge  to  form  them  comes  from  the 
definition  of  the  task,  plus  general  skills  in  reading  and  writing.  Comprehend  is  an  expectation-based 
scheme  that  associates  both  syntactic  and  semantic  knowledge  with  individual  words.  It  constructs  an 
initial  (possibly  incomplete)  model;  it  also  leaves  as  a  byproduct  a  model  of  each  premise  as  a 
proposition,  with  parts  subject,  object  and  sign  (the  predicate),  and  quantifier.  Prop-to-prop, 
Model-to-prop,  and  Prop-to-model  have  operators  required  to  manipulate  models  of  situations  and 
models  of  propositions,  as  well  as  attention  operators  to  instantiate  the  manipulations. 
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The  Behavior  of  the  System 


Figure  4  illustrates  the  system’s  behavior.  (1)  It  starts  in  Syllogism  and  applies  read-premise, 
implemented  in  Comprehend,  to  the  hist  and  then  the  second  premise.  (2)  This  results  in  an  initial 
model,  plus  the  two  internal  propositions.  This  encoding  only  extracts  information  about  the  subject  of 
the  premise.  (3)  Since  the  overall  task  is  to  produce  a  conclusion,  build-conclusion  is  applied.  Its  space 
(Build-conclusion)  puts  together  legal  propositions.  The  task  decomposes  into  discovering  the  subject, 
predicate  and  quantifier  of  the  conclusion.  Task  knowledge  permits  determining  some  parts  without  other 
parts  being  specified.  Incomplete  or  incorrect  knowledge  leads  to  composing  invalid  conclusions. 

©Initial  Sum  ,  archen  are 

(are  not  bowien J  (7  J  V  \  bamlat  J  jj]  canoeists 

VST  £*>  + 

K  f  .  /  o  AN/Some  trchenS  I 

I  h£5A.V  /  I  t Trfci*  \  «ra  no*  bowlery 

Irapanet  \7  J  Impeue  |  'Or  /  ^  V  Impanel 


Initial  Siam 


archen  are 


gr-~,n=!>-n  + 


□  =£> 


I  (ij (i) ~~ 

□  (sublet)  =£>  Q  (predate)  ={>□  (quantifier)  -i.  I  I  arehan  are  + 

A’d>m  A'3S™  Impure  |  emomu  /jj\ 

Impeue  canoeist*  ^^UK*>ro,>  |  |(anead)  (copy  Q  + 
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©  © 

©  @ 
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Figure  4:  Behavior  on  Some  archers  are  not  bowlers,  AU.  canoeists  are  bowlers. 


i  i  (cna*°  rn  + 

1 — 1  auxiliary) 


(4)  Generating  the  subject  is  tried  first,  which  uses  Prop-to-prop  because  the  propositions,  not  the 
model,  distinguish  between  subjects  and  objects.  (3)  Attcnd-to-prop  selects  the  first  proposition  and 
copy-subject  creates  the  subject  of  the  conclusion  (archers).  (6)  Next,  genera te-predicate  is  selected, 
which  uses  Model-to-prop,  because  the  propositions  contain  no  useful  information  about  the  predicate. 
(7)  The  attend-to-object  operator  applies,  but  no  others,  because  the  model  is  incomplete.  This  leads  to 
augmenting  the  model,  using  Prop-to-model.  (8)  Attend-to-prop  selects  premises  to  extract  more 
information,  but  neither  premise  yields  anything.  (9)  Create-auxiliary  produces  a  new  proposition  in 
Prop-to-prop.  It  attends  to  the  second  premise  and  applies  operators  which  convert  it,  creating  the  new 
premise  All  bowlers  are  canoeists.  (10)  This  allows  solving  in  Prop-to-model  to  resume,  by  focusing 
attention  on  this  new  proposition  and  using  it  to  augment  the  model.  (11)  The  model  now  suggests  a 
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predicate,  so  solving  is  able  to  continue  i.1  Model-to-prop  to  obtain  the  predicate  for  the  conclusion  ( are 
canoeists).  (12)  All  that  remains  in  Build-conclusion  is  to  generate  the  quantifier.  The  model  does  not 
represent  quantifiers,  so  Prop-to-prop  is  used  again.  (13)  It  attends  to  the  first  premise  and  copies  its 
quantifier  (some),  finally  obtaining.  Some  archers  are  canoeists.  This  is  incorrect,  but  many  humans  fail 
this  syllogism  as  welL  Correctness  depends  on  knowledge  being  available  at  many  local  choices. 


Human  Data  and  Soar  Performance 

Figure  5  presents  data  from  (Johnson-Laiid  &  Bara,  1984)  by  20  University  of  Milan  students  on  all  64 
syllogisms  (with  unlimited  time)  and  also  the  responses  by  Soar.  The  four  sections  of  the  chart 
correspond  to  the  four  figures  (Figure  1, right).  Each  row  corresponds  to  one  of  the  9  legal  responses. 
The  top  number  in  each  cell  indicates  the  number  of  subjects  giving  that  response  to  a  particular 
syllogism.  Some  archers  are  not  bowlers  and  AU  canoeists  are  bowlers  (Figure  4)  is  abbreviated 
Oxy,Azy,  and  occurs  in  the  lower  left  quadrant,  where  we  see  that  8  subjects  responded  Ixz  (Some 
archers  are  canoeists),  7  responded  Oxz  (Some  archers  are  not  canoeists),  3  responded  NVC  (no  valid 
conclusion)  and  2  subjects  gave  illegal  responses.  Valid  responses  are  shaded  (Oxz  for  7/20  correct). 
Only  38%  percent  of  all  responses  were  correct  and  7  syllogisms  were  solved  by  no  one. 

Individual  humans  behave  differently  from  each  other  and  from  themselves  over  time,  due  to  learning 
and  other  factors.  The  data  of  Figure  5  are  a  composite,  as  shown  by  multiple  responses.  A  family  of 
Soar  systems  is  required  to  correspond  to  this  human  variation.  We  varied  the  theory  along  3  dimensions: 
(1)  whether  auxiliary  propositions  are  created,  as  in  our  example  (2  choices);  (2)  how  premises  augment 
objects  with  not  annotations  (3  choices);  and  (3)  whether  premises  about  some  x  augment  objects  about  x 
(2  choices).  The  first  dimension  is  one  of  reasoning  power,  the  other  two  involve  the  semantics  of 
interpreting  premises.  These  dimensions  form  a  family  of  12  variants. 

This  small  family  accounts  for  980  out  of  1154  (85%)  observed  legal  responses  (126/1280  responses 
were  illegal  and  not  recorded)  by  covering  131  out  of  the  193  cells  (68%)  that  contain  1  or  more 
responses  (all  cells  with  more  than  6  subjects  are  predicted  with  one  exception  [Oyx.  Ayz  =  Izx]).  Only 
one  response  is  predicted  that  is  not  given  by  any  subject  [Oyx,  Ayz  =  Ozx).  Frequencies  were  assigned 
to  the  different  members  of  the  family  to  produce  the  fit  shown  in  parentheses  in  Figure  5  (15/20  subjects 
were  assumed  in  the  family  since  23%  of  responses,  many  illegal,  were  unpredicted).  No  simple  measure 
of  fit  is  available,  but  the  correlation  between  subjects  and  systems  is  .87. 

The  theory  produces  the  classical  effects,  such  as  the  atmosphere  effect  (Woodworth  &  Sells.  1935).  the 
conversion  hypothesis  (Chapman  &  Chapman,  1959)  and  the  figural  effect  (Johnson-Laiid,  1983).  Space 
does  not  permit  showing  the  analysis,  but  they  need  only  be  traced  out  in  Figure  5.  The  atmosphere  and 
figural  effects  arise  because  the  syntactic  form  of  the  premises  serves  as  search  control  in  the  construction 
of  the  conclusion.  The  conversion  effect  arises  when  this  search  control  is  insufficient  and  a  new 
proposition  is  created. 

According  to  the  theory,  there  are  three  main  sources  of  difficulty:  (1)  making  unwarranted 
assumptions  about  the  premises;  (2)  failing  to  consider  ail  the  implicit  ramifications  of  the  premises;  and 


Figure  5:  Data  (from  Johnson- Laird  &  Bara.  1984)  and  Soar  predictions  in  ( ). 

(3)  failing  to  consider  ail  the  possible  conclusions  based  on  a  (possibly  correct)  model.  Syllogisms  are 
difficult  to  the  extent  they  present  opportunities  for  these  processing  difficulties  (e.g..  have  implicit 
ramifications  relevant  to  the  conclusions).  This  predicts  that  better  subjects  will  extract  more  information 
from  the  premises  without  making  unwarranted  assumptions  or  that  they  will  search  for  conclusions  more 
extensively. 


We  designed  a  family  of  systems  based  on  10  parameters,  which  includes  the  current  3-parameter 
family,  with  the  values  (mostly  binary)  of  each  parameter  being  independently  ordered  by  validity  (so  that 
better  values  correspond  to  more  powerful  and  correct  ways  of  building  models).  When  all  parameters 
take  on  their  optimal  values,  perfect  performance  should  occur.  Better  solvers  should  occur  within  this 
space  with  interpretable  parameter  settings.  To  test  this,  we  analyzed  another  set  of  20  subjects  58%  of 
whose  responses  were  correct  (Johnson-Laird  &  Steedman,  1978).  We  implemented  a  small  sub-family 
(24  variants  including  the  12)  that  covered  87%  of  the  responses  and  67%  of  the  ceils;  it  did  however 
predict  1 1  responses  not  given  by  any  subjects.  The  parameter  settings  of  the  modal  system  for  the  new 
distribution  are  better  (higher  in  validity  ordering)  than  those  of  the  old  distribution’s  modal  system  on  3 
parameters  and  the  same  on  the  other  7. 
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The  explanatory  power  of  this  theory  appeals  better  than  existing  theories.  Their  predictions  are  less 
accurate  in  that  they  predict  a  large  number  of  responses  that  were  not  observed  in  any  subjects  and  they 
do  not  make  strong  frequency  predictions.  Most  theories  only  explain  highly  aggregate  data.  However, 
the  data  used  here  (Figure  S)  is  still  aggregated  over  subjects,  and  nothing  has  yet  been  done  with  timing 
and  protocol  data.  So  ample  opportunity  remains  to  challenge  and  improve  the  present  theory. 

This  theory  has  much  to  recommend  it  generally.  It  predicts  flexible  activity,  e.g.,  going  back  to  the 
premises  to  try  to  extract  more  information.  Its  spaces  (especially  executive  ones)  are  substantially  less 
arbitrary  than  prior  simulations  (e.g.,  Comprehend  embodies  a  theory  of  elementary  language 
comprehension).  Although  not  reported  on  here,  the  present  theory  involves  a  theory  of  learning,  which 
is  an  essential  part  of  any  general  account  of  human  cognitive  behavior.  These  attributes  and  others  anse 
primarily  horn  this  theory  of  syllogism  being  embedded  in  Soar  as  a  unified  theory  of  cognition. 
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